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OBJECT DETECTION SYSTEM AND OBJECT DETECTION METHOD 



BACKGROUND OF THE INVENTION 

1. Field of Invention 

5 [0001] The invention relates to an object detection system that detects an object using 

radar and images. 

i 

2. Description of Related Art 

[0002] Recently an operation support system, for example, a collision avoidance control 
system, an adaptive cruise control system, a tracking control system and the like has been 

10 increasingly developed. The detection of an object (obstruction) such as a preceding vehicle is 
essential for the operation support system. An object detection system disclosed in 
publications JP-A-2003-84064 and JP-A-7-125567 as below includes two detection units, for 
example, a radar such as a laser radar, and a stereo camera that shoots an image. The object 
detection system performs collation between detection results of the radar and the detected 

1 5 image. The preceding object is detected based on the aforementioned collation results. 

[0003] In the generally employed object detection system using two kinds of detection 
units, the respective detection results of those detection units each obtained instantaneously are 
collated Deterioration in accuracy of at least one of those detection units may interfere with 
accurate collating operation. The detection accuracy of the aforementioned object detection 

20 system is likely to be deteriorated even if two different detection units are employed. 

Accordingly, such system may mistakenly detect the object or foil to detect the object 



SUMMARY OF THE INVENTION 
[0004] It is an object of the invention to provide an object detection system capable of 
25 detecting an object with high accuracy. 

[0005] An object detection system according to an aspect of the invention includes a radar 
detection means that detects an object using a radar, an image detection means that detects an 
object using an image, and a collating means that performs collation between a detection result 
of the radar detection means and a detection result of the image detection means so as to 
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determine whether an identical object is detected by the radar detection means and the image 
detection means. The collating means performs a collation between an object detected by the 
radar detection means in a present collation and an object that has been determined as being 
detected by the radar detection means and the image detection means in a previous collation, 
and further performs a collation between an object detected by the image detection means in a 
present collation and an object that has been determined as being detected by the radar 
detection means and the image detection means in the previous collation when it is determined 
that the identical object is detected by the radar detection means and the image detection means 
in the previous collation. Then the collating means determines whether the radar detection 
means and the image detection means detect the identical object based on the first and the 
second collations. 

[0006] According to another aspect of the invention, in a method of detecting an object in 
an object detection system including a radar detection means that detects an object using a 
radar, an image detection means that detects an object using an image, and a collating means 
that performs collation between a detection result of the radar detection means and a detection 
result of the image detection means so as to determine whether an identical object is detected 
by the radar detection means and the image detection means, a first collation is performed 
between an object detected by the radar detection means in a present collation and an object 
that has been determined as being detected by the radar detection means and the image 
detection means in a previous collation, and a second collation is performed between an object 
detected by the image detection means in a present collation and an object that has been 
determined as being detected by the radar detection means and the image detection means in 
the previous collation when it is determined that the identical object is detected by the radar 
detection means and the image detection means in the previous collation. It is then determined 
whether the radar detection means and the image detection means detect the identical object 
based on the first and the second collations. 

[0007] In die object detection system and object detection method, the object is detected 
by the radar detection means and the image detection means. The collating means of the object 
detection system and object detection method performs collation between detection results of 
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the radar detection means and detection results of the image detection means at a 
predetermined time interval. If it is determined that the same object is detected by the 
aforementioned two detection units based on the results of the previous collation, the present 
collation is performed using the previous collation results. More specifically, the collating 
5 . means performs collation between the object detected by the two detection units in the 

previous collation and the objects presently detected by the radar detection means, and between 
the object detected by the two detection units in the previous collation and the obj ects presently 
detected by the image detection means. Then it is determined whether the identical object is 
detected by those two detection units based on results of collation of the previous collation 

1 0 results with those of the object presently detected by the radar detection means and the image 
detection means. The detection results of the two detection units are not directly collated with 
each other, but each detection results of the two detection units is time-series collated using the 
previous collation results. This may improve the detection accuracy and stabilize the detection 
compared with the generally employed object detection system in which instantaneous 

1 5 detection results are collated. Even if the accuracy of at least one of those two detection units is 
deteriorated instantaneously, the collation may be performed using the previous collation 
results. Accordingly the detection results of the two detection units may be easily collated. 

[0008] A millimeter-wave radar, laser radar and the like may be used as the radar 
detection means. A stereo camera may be used as the image detection means. 

20 [0009] In the object detection system and object detection method according to the aspect 

of the invention, the collating means performs a collation between objects detected by the radar 
detection means in the present detection, which are obtained by excluding the object 
determined as have been detected by the radar detection means and the image detection means, 
and objects detected by the image detection means in the present detection, whiclx are obtained 

25 by excluding the object determined as have been detected by the radar detection means and the 
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image detection means such that it is determined whether the identical object is detected by the 

radar detection means and the image detection means. 

[0010] In the case where it is determined that the identical object is detected by the two 
.. detection units in the present collation using the previous collation results, the determined 
5 object is excluded from the objects presently detected by the radar detection means and the 

image detection means, respectively so as to be collated. It is determined whether there is the 
identical object among those presently detected by the two detection units. In the object 
detection system and object detection method, instantaneous collation is performed between 
the two detection units that have been instantaneously obtained by the two detection units in 
10 consideration with the time-series collation results. Accordingly the object detection system 
makes it possible to collate objects that have been presently detected reliably. 

[0011] The object detection system and object detection method according to the 
invention performs the time-series collation using the previous collation results so as to detect 
the object with high accuracy. 

15 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0012] The foregoing and further objects, features and advantages of the invention will 
become apparent from the following description of preferred embodiments with reference to 
the accompanying drawings, wherein like numerals are used to represent like elements and 
20 wherein: 

Fig. 1 is a view that shows a structure of an obstruction detection system as an 
embodiment according to the invention; 

Fig. 2 is an explanatory view representing the time-series collation between the 
previous fusion objects and the present millimetric-wave objects and between the previous 
25 fusion objects and the present image objects in the obstruction detection system as shown in 
Fig. 1; 
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Fig. 3 is an explanatory view representing the instantaneous collation between the 
millimetric-wave objects and the image objects each obtained by excluding the time-series 
fusion objects therefrom in the obstruction detection system as shown in Fig. 1; 

Fig. 4 is a flowchart representing a collating process executed in the obstruction 
5 detection system as shown in Fig. 1; 

Fig. 5 is a flowchart representing a time-series collating process executed in the 
obstruction detection system as shown in Fig. 1; and 

Fig. 6 is a flowchart representing an instantaneous collating process executed in the 
obstruction detection system as shown in Fig. 1. 
10 DETAILED DESCRIPTION OF PREFERRED EMBODIMENT 

[0013] An embodiment of the object detection system according to the invention will be 
described referring to the drawings. 

[0014] In this embodiment, the object detection system according to the invention is 
applied to an obstruction detection system provided in a vehicle so as to detect an obstruction 
15 that precedes the vehicle. The obstruction detection system in this embodiment is provided 
with two detection units, that is, a millimeter-wave radar and a stereo camera. 

[0015] Referring to Fig. 1, an obstruction detection system 1 will be described Fig. 1 
shows a structure of the obstniction detection system according to this embodiment 
[0016] The obstruction detection system 1 is provided in a vehicle to detect an 
20 obstruction, that is, a running vehicle or the like that precedes the vehicle. The obstruction 
detection system 1 functions in providing obstruction information to the operation support 
system, for example, a collision avoidance control system, an adaptive cruise control system, a 
tracking control system and the like, which requires the information with respect to the 
preceding obstruction. In the obstruction detection system 1, each detection results of the two 
25 detection units is time-series collated with the previous collation results. Then instantaneous 
collation is performed using the present detection results obtained by excluding the time-series 
collation results therefrom such that the obstruction which precedes the vehicle is identified 
with high accuracy. The obstruction detection system 1 is provided with a millimeter-wave 
radar 2, a stereo camera 3, and an ECU 4 (Electronic Control Unit). The obstruction detection 
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system 1 may be provided independently from the operation support system. That is, it may be 
structured to transmit the detected obstruction information to the operation support system. 
Alternatively the obstruction detection systeml may be assembled with the operation support 
system. 

5 [0017] In this embodiment, the milJtimeter-wave radar 2 corresponds with the radar 

detection means, the stereo camera 3 corresponds with the image detection unit, and die ECU 4 
corresponds with the collating unit 

[0018] In this embodiment, a millimetric-wave object refers to an object detected by the 
millimeter-wave radar 2, and an image object refers to an object detected by the stereo camera 

10 3. A fusion object refers to the object identified as the one that has been detected by the 

millimeter-wave radar 2 and the stereo camera 3 through collation between the millimetric- 
wave objects and the image objects, and serves as the obstruction information supplied by the 
obstruction detection system 1 . An independent millimetric-wave object represents the object 
that is detected only by the millimeter-wave radar 2. In other words, the independent 

15 millimetric-wave object is obtained by excluding the fusion object from the millimetric-wave 
objects. An independent image object represents the object that is detected only by the stereo 
camera 3. In other words, the independent image object is obtained by excluding the fusion 
object from the image objects. The respective objects exhibit the information with respect to 
the distance between the vehicle and the preceding obstruction, the relative speed of the 

20 obstruction with respect to the vehicle speed, the angle defined by the obstruction and the 
running direction of the vehicle (information of the lateral position) for determining the 
position of the obstruction with respect to the vehicle. 

[0019] The millimeter-wave radar 2, radar for detecting an object using millimetric wave, 
is attached in the center of a front surface of the vehicle. The millimeter-wave radar 2 scans the 

25 millimetric wave on the horizontal plane so as to be emitted from the vehicle forward, and 

receives the reflected millimetric wave. The millimeter-wave radar 2 measures the time period 
elapsing from the emission to the receipt of the millimetric wave such that the distance from 
the front end of die vehicle to the preceding object is calculated. Hie millimeter-wave radar 2 
further calculates a relative speed of the vehicle with respect to that of the preceding object 
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using Doppler Effect The millimeter-wave radar 2 detects the direction of the millimetric 
wave that reflects most intensely, based on which an angle defined by the traveling direction of 
the vehicle and that of the preceding object is calculated. The millimeter-wave radar 2 is 
capable of detecting the object upon receipt of the reflecting millimetric wave. At every receipt 
5 . of the reflecting millimetric wave, therefore, one millimetric object is obtained The 

millimeter-wave radar 2 serves to calculate the distance, the relative speed, and the angle. 
However, die ECU 4 may be structured to calculate those values based on detection results of 
the millimeter wave radar 2. 

[0020] The millimeter-wave radar 2 is capable of detecting the distance and the relative 

10 speed with relatively higher accuracy, but detecting the angle with relatively lower accuracy. 
As the millimeter-wave radar 2 calculates the distance based on the time elapsing from 
emission of the millimetric wave to reflection thereof, the accuracy in the calculated distance is 
relatively higher. As the relative speed is calculated using Doppler effect, the resultant value of 
the relative speed exhibits high accuracy. The millimeter-wave radar 2 fails to identify the 

1 5 point at which the millimetric wave reflects most intensely in the width direction of the object 
Therefore, the position in the width direction (lateral position) is likely to fluctuate, reducing 
accuracy in the angle. 

[0021] The stereo camera 3 includes two sets of CCD cameras (not shown), which are 
arranged apart at a distance of approximately several 10 cms in the horizontal direction. The 

20 stereo camera 3 is also attached in the center of the front surface of the vehicle. The stereo 
camera 3 transmits the respective image data shot by those two CCD cameras to an image 
processing portion (not shown). The image processing portion may be assembled with die 
stereo camera 3 or formed within the ECU 4. 

[0022] The image processing portion identifies the object based on the respective image 

25 data, and obtains the positional information with respect to the object. The stereo camera 3 is 
capable of detecting the object when the object is identified based on two image data. At every 
identification of the object, one image object is obtained. The image processing portion 
calculates the distance from the front end of the vehicle to the preceding object by triangulation 
using the difference in views of the object between two image data The image processing 
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portion calculates the relative speed based on the change in the calculated distance as an elapse 
of time. The image processing portion detects both ends of the detected object in the width 
direction such that each angle defined by the traveling direction of the vehicle and the 
respective ends of the object is calculated. Accordingly the lateral positional information of the 
5 . image object includes two types of angle information with respect to both ends of the object in 
the width direction. 

[0023] Each detection result ofthe stereo camera 3 with respect to the distance and the 
relative speed exhibits relatively lower accuracy but the detection result with respect to the 
angle exhibits relatively higher accuracy. As both ends of the object in the width direction can 

10 be detected with high accuracy based on the left and right image data, the detection result with 
respect to the angle may exhibit high accuracy. However, as the image data are supplied from 
left and right CCD cameras several 10 cms apart, the triangulation is performed at substantially 
acute angle for calculating the distance. Accordingly, each accuracy of the distance and the 
relative speed may be reduced. 

15 [0024] The ECU 4, that is, electronic control unit, includes a CPU (Central Processing 

Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The 
ECU 4 is connected with the millimeter-wave radar 2 and the stereo camera 3. The ECU 4 
receives data of the millimetric-wave object from the millimeter-wave radar 2, and the image 
object from the stereo camera 3. Then the ECU 4 performs collation between the previous 

20 collation results (previous fusion objects) and the present millimetric-wave objects and the 
image objects received in the present collation, respectively at a predetermined time interval 
based on the master clock of the CPU so as to obtain the present fusion objects. When the 
image processing portion is provided in the ECU 4, the image data are received by the ECU 4 
from the stereo camera 3 such that the image object is derived from the image data. 

25 [0025] The ECU 4 executes the time-series collation between the previous fusion objects 

(n3 pre), and present millimetric-wave objects (n m) and present image objects (n i) both 
derived from the present collation, respectively so as to detect time-series fusion objects (n3')« 
The ECU 4 then executes the instantaneous collation between the millimetric-wave objects (n 
m-n3') obtained by excluding the time-series fusion objects, and the image objects (n i-n3') 



WO 2005/024460 PCT/IB2004/002927 

9 

obtained by excluding the time-series fusion objects so as to detect instantaneous fusion objects 
(n3"). The time-series fusion objects (n3') are added to the instantaneous fusion objects (n3") 
to set the present fusion objects (n3 = n3'+n3") in the present collation. The ECU 4 further 
sets the present millimetric-wave objects which have not been selected as the fusion object to 
5 independent millimetric-wave objects (nl = n m-n3), and the present image objects which have 
not been selected as the fusion object to independent image objects (n2 = n i-n3). 

[0026] Referring to Fig. 2, the time-series collation executed in the ECU 4 will be 
described. Fig. 2 represents the time-series collation between the previous fusion objects, and 
the present millimetric-wave objects and the present image objects, respectively in an 
10 obstruction detection system. In an example shown in Fig. 2, three previous fusion objects FF1 
to FF3 are obtained in the previous collation, four present millimetric-wave objects NM1 to 
NM4 are obtained in the present detection performed by the millimeter-wave radar 2, and five 
present image objects Nil to NI5 are obtained in the present detection performed by the stereo 
camera 3. 

15 [0027] In the case where n3 pre previous fusion objects are obtained in the previous 

collation, the ECU 4 executes collation between the previous fusion objects with high 
possibility of being detected as the fusion object in the present collation, and n m millimetric- 
wave objects detected by the millimeter-wave radar 2 in the present detection and n i image 
objects detected by the stereo camera 3 in the present detection, respectively. 

20 [0028] The ECU 4 fetches each of those n3 pre previous fusion objects so as to be 

collated with those n m present millimetric-wave objects sequentially such that it is determined 
whether there is the present millimetric-wave object with high similarity to the fetched 
previous fusion object In the present collation, the similarity of each of the n m present 
millimetric-wave objects to each of the previous fusion objects is calculated using the formula 

25 for obtaining similarity of the millimetric-wave objects. The present millimetric-wave object 
with the highest similarity is then selected The millimetric-wave similarity formula is used to 
calculate the similarity by weighting those values of the distance, relative speed, and angle of 
the respective present millimetric-wave objects. In the formula, the distance and the relative 
speed each detected by die millimeter-wave radar 2 with higher accuracy are heavily weighted, 
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and the angle with lower accuracy is lightly weighted. If the similarity value of the selected 
present millimetric-wave object is equal to or higher than a threshold value (used for 
determining the similarity, based on which coincidence in the object is determined), it is 
determined that the present millimetric-wave object is identical to the fetched previous fusion 

5 object As a result of the millimetric-wave object collation, the previous fusion object is 
marked. The ECU 4 executes the collation of the present millimetric-wave objects with 
respect to the previous fusion objects repeatedly by n3 pre times. 

[0029] hi the example shown in Fig. 2, there is the present millimetric-wave object 
having the higher similarity to the previous fusion object FF1 . As a result of the mfllimetric- 

10 wave object collation, the previous fusion object EF1 is marked. Also there is the present 

millimetric-wave object having the higher similarity to the previous fusion object FF2. As a 
result of the millimetric-wave object collation, the previous fusion object FF2 is marked. There 
is no present millimetric-wave object having higher similarity to die previous fusion object 
FF3. As a result of the millimetric-wave object collation, the previous fusion object FF3 is not 

15 marked 

[0030] Each of the n3 pre previous fusion objects is fetched so as to be collated with the n 
i present image objects sequentially in the ECU 4, and it is determined whether there is the 
present image object having higher similarity to the fetched previous fusion object In the 
aforementioned collation, each similarity of the n i present image objects to the previous fusion 

20 objects is calculated using the image similarity formula so as to select fee present image object 
having the highest similarity. The image similarity formula is used to calculate the similarity 
by weighting those values of the distance, relative speed, and angle of the respective present 
image objects. In the formula, the angle detected by the stereo camera 3 having higher 
accuracy is heavily weighted, and the distance and the relative speed with lower accuracy are 

25 ligjhtly weighted. If the similarity value of die selected present image object is equal to or 

higher than the threshold value, it is determined that the present image object is identical to the 
fetched previous fusion object As a result of the image object collation, the previous fusion 
object is marked. The ECU 4 executes the collation of the present image objects with respect 
to the previous fusion objects repeatedly by n3 pre times. 
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[0031] Referring to the example shown in Kg. 2, there is the present image object having 
higher similarity to the previous fusion object FF1, the previous fusion object FF1 is marked as 
a result of the image object collation. There is no present image object having higher similarity 
to the previous fusion object FF2, the fusion object FF2 is not marked. There is the present 
5 image object having higher similarity to the previous fusion object FF3, the previous fusion 
object FF3 is marked. 

[0032] The ECU 4 then performs comparison of the results between the millimetric-wave 
object collation and the image object collation so as to select the previous fusion object that is 
marked in both results. The selected previous fusion object is set as the time-series fusion 

10 object in which the information of the distance and the relative speed is set based on the data of 
the present millimetric-wave object and the information of the angle is set based on the data of 
the present image object. The number of the time-series fusion objects is equal to or smaller 
than that of the previous fusion objects. 

[0033] Referring to the example shown in Fig. 2, in the results of the millimetric-wave 

15 object collation, the previous fusion objects FF1 and FF2 are marked with circle. In the results 
of the image object collation, the previous fusion objects FF1 and FF3 are marked with square. 
As a result, only the previous fusion object FF1 is selected as the time-series fusion object 

[0034] The instantaneous collation executed in the ECU 4 will be described referring to 
Fig. 3. Fig. 3 represents the instantaneous collation between the millimetric-wave objects 

20 except the time-series fusion objects and the image objects except the time-series fusion objects 
in the obstruction detection system. Referring to the example of Fig. 3, five millimetric-wave 
objects Ml to M5 are detected by the millimeter-wave radar 2 except the time-series fusion 
objects, and six image objects II to 16 are detected by the stereo camera 3 except the time- 
series fusion objects. 

25 [0035] In the ECU 4, n3' time-series fusion objects are excluded from n m present 

millimetric-wave objects derived from the detection results of the millimeter-wave radar 2. 
Each one of the present millimetric-wave objects (n m-n3') is fetched as a reference for the 
purpose of performing die collation with respect to the millimetric-wave object Then n i-n3' 
image objects are collated with respect to the fetched millimetric-wave object sequentially, and 
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only one image object that is closest to the millimetric-wave object is selected. In this case, 
collation is executed with respect to the distance between the object and the vehicle, and the 
angle defined by the object and the traveling direction of the vehicle as well as the relative 
speed of the object with respect to the vehicle if required. In case of the distance collation, the 
5 . distance difference (for exairple, several meters) is set as the threshold value in accordance 
with the accuracy of the millimeter-wave radar 2. When the difference between the distance 
from the millimetric-wave object to the vehicle and the distance from the closest image object 
to the vehicle is equal to or larger than the threshold value, the closest image object cannot be 
selected. In case of the angle collation, the angle difference (for example, several degrees) is 

10 set as the threshold value in accordance with the accuracy of the millimeter- wave radar 2. 
When the difference between the angle defined by the millimetric-wave object and the 
traveling direction of the vehicle, and the angle defined by the closest image object and the 
traveling direction of the vehicle is equal to or larger than the threshold value, the closest image 
object cannot be selected. When the closest image object is selected, the millimetric-wave 

15 object and the selected image object as being closest thereto are stored as the millimetric-wave 
base pair in the ECU 4. The ECU 4 executes the aforementioned collation with respect to the 
millimetric-wave objects repeatedly by 
n m-n3' times. 

[0036] In the example shown in Fig. 3, when the millimetric-wave object Ml is fetched 
20 as the reference, an image object II is selected such that a millimetric-wave base pair MP1 is 

determined. When the millimetric-wave object M2 is fetched as the reference, an image object 
12 is selected such that a millimetric-wave base pair MP2 is determined. When a millimetric- 
wave object M3 is fetched as the reference, an image object 12 is selected such that a 
millimetric-wave base pair MP3 is determined When the millimetric-wave object M4 is 
25 fetched as the reference, any one of the image objects II to 16 cannot be selected to form the 

millimetric-wave base pair because each distance between the respective image objects II to 16 
and the millimetric-wave object M4 exceeds the threshold value. When the millimetric-wave 
object M5 is fetched as the reference, an image object D is selected such that a millimetric- 
wave base pair MP4 is determined 
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[0037] In the ECU 4, the n i present image objects are derived from detection results of 
the stereo camera 3. The n3' time-series fusion objects are excluded from those present image 
objects (n i-n3'), each of which is fetched such that the collation with respect to the image 
object is executed. Each of the millimetric-wave objects (n m-n3') is collated with the fetched 
5 image object as the reference sequentially so as to select only one millimetric-wave object that 
is closest to the reference image object. In this case, like the collation with respect to the 
millimetric-wave objects, collation is perfonned with respect to the distance between the object 
and the vehicle, and the angle defined by the object and the traveling direction of the vehicle. 
Each difference in the aforementioned distance and the angle is set as the threshold value in 

10 accordance with the accuracy in the stereo camera 3. In the ECU 4, in the case where the 

difference in the distance between the reference image object and the millimetric-wave object 
closest thereto is equal to or larger than the threshold value, or the difference in the angle is 
equal to or larger than the threshold value, the millimetric-wave object considered as being 
closest to the reference image object cannot be selected. When the closest millimetric-wave 

15 object is selected, die reference image object and the millimetric-wave object closest thereto 
are stored in the ECU 4 as an image object base pair. The aforementioned collation with 
respect to the image objects is repeatedly executed by n i-n3' times. 

[0038] In the example shown in Fig. 3, when the image object II is fetched as the 
reference, the millimetric-wave object Ml is selected such that an image base pair DPI is 

20 determined. When the image object J2 is fetched as the reference, the millimetric-wave object 
M2 is selected such that an image base pair JP2 is determined. When the image object D is 
fetched as the reference, the millimetric-wave object M5 is selected such that an image base 
pair IP4 is determined. When the image target object 14 is fetched as the reference, the 
millimetric-wave object M5 is selected such that an image base pair IP4 is determined When 

25 the image object 15 is fetched as the reference, any one of the millimetric-wave target objects 
Ml to M5 cannot be selected to form the image base pair because each difference in the 
distance between the respective millimetric-wave objects Ml to M5, and the image object 15 
exceeds the threshold value. When the image object 16 is fetched as the reference, any one of 
the millimetric-wave target objects Ml to M5 cannot be selected to form the image base pair 
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because each difference in the distance between the respective millimetric-wave objects Ml to 
M5, and the image object 16 exceeds the threshold value. 

[0039] The ECU 4 performs comparison between the millimetric-wave base pairs and the 
image base pairs sequentially so as to select the millimetric-base pair and the image base pair 
5 . each including the identical millimetric-wave object and the image object The ECU 4 further 
sets the selected combination of the millimetric-base pair and the image base pair each 
including the identical millimetric-wave object and the image object as a fusion pair 
(instantaneous fusion object). Then the information with respect to the distance and the relative 
speed derived from the millimetric-wave object data, and the information with respect to the 

10 angle derived from the image object data is set as the fusion object information. 

[0040] In the example shown in Fig. 3, each of the millimetric-wave base pair MP1 and 
the image base pair IP1 includes the identical millimetric-wave object Ml and the image object 
II , thus forming the fusion pair FP 1 . Each of the millimetric-wave base pair MP2 and the 
image base pair JP2 includes the identical millimetric-wave object M2 and the image object 12, 

15 thus forming the fusion pair FP2. Each of the millimetric-wave base pair MP4 and the image 
base pair IP3 includes the identical millimetric-wave object M5 and the image object D, thus 
forming the fusion pair FP3 . The millimetric-wave base pair MP3 has no image base pair 
including the identical millimetric-wave object and the image object The image base pair IP4 
has no millimetric-wave base pair including the identical millimetric-wave object and the 

20 image object 

[0041] The collation process performed in the obstruction detection system 1 will be 
described referring to Fig. 1 . The collation process in general will be described referring to the 
flowchart of Fig. 4. The time-series collation process will be described referring to the 
flowchart of Fig. 5, and the instantaneous collation process will be described referring to the 

25 flowchart of Fig. 6, respectively. Kg. 4 is the flowchart representing the collation process 

performed in the obstruction detection system. Fig. 5 is the flowchart representing the time- 
series collation process performed in the obstruction detection system. Fig. 6 is the flowchart 
representing the instantaneous collation process performed in the obstruction detection system. 
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[0042] The collation process in general will be described referring to the flowchart of Fig. 
4. In the obstruction detection system 1, in step SI, millimetric-wave objects (n m) are 
intermittently detected by the millimeter-wave radar 2, and image objects (n i) are also 
intermittently detected by the stereo camera 3. In the obstruction detection system 1, the 

5 . collation is executed at a predetermined time interval. If the result of the previous collation 
shows detection of the previous fusion objects (n3 pre), the previous fusion objects are time- 
series collated with respect to the present millimetric-wave objects (n m) and the present image 
objects (n i), respectively so as to determine the time-series fusion objects (n3'). As the 
number of the previous fusion objects in the first collation is zero, the time-series collation is 

10 not executed. When the number of the previous fusion objects is zero in the case other than the 
first collation, the time-series collation is not executed. 

[0043] In step S2, the instantaneous collation is executed between the millimetric-wave 
objects derived from excluding the time-series fusion objects from the present millimetric- 
wave objects (n m-n3') and the image objects derived from excluding the time-series fusion 

15 objects from the present image objects (n i-n3') so as to determine the instantaneous fusion 
objects (n3* Jf the number of the time-series fusion objects is zero, the instantaneous 
collation is performed between the present millimetric-wave objects (n m) and the present 
image objects (n i). If it is determined that each of the present millimetric-wave object and the 
present image object is identical to the previous fusion object, the instantaneous collation is not 

20 executed. 

[0044] In the obstruction detection system 1, in step S3, the number of the time-series 
fusion objects is added to that of the instantaneous fusion objects to determine the fusion 
objects in the present collation (n3 = n3' + n3")- Further the independent millimetric-wave 
objects (nl = n m-n3) and the independent image objects (n2 = n i-n3) are determined 
25 respectively. In the obstruction detection system 1, the fusion objects, independent millimetric- 
wave objects, and image objects are determined at every collation process. The determined 
fusion objects serve as the previous fusion objects in the subsequent collation process. 

[0045] The time-series collation process will be described referring to the flowchart of 
Fig. 5. In the obstruction detection system 1, it is determined whether the fusion object has 
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been detected in the previous collation. If it is determined that the fusion object has been 
detected in the previous collation, the control proceeds to the following process. 

[0046] In step S 10, each similarity of n m present millimetric-wave objects to the 
previous fusion objects (n3 pre) weighted on the distance and the relative speed is calculated in 
5 the obstruction detection system 1. It is determined whether there is the millimetric-wave 
object having higher similarity (equal to or larger than the threshold value) to the previous 
fusion object If it is determined that there is the millimetric-wave object having higher 
similarity to the previous fusion object, the previous fusion object in reference to the 
millimetric-wave collation results is marked. 

1 0 [0047] Then in step S 1 1 , it is determined whether all the detennination with respect to n3 

pre previous fusion objects has been completed. This step is executed repeatedly until 
completion of the determination. In the obstruction detection system 1, the previous fusion 
object that has been detected in the previous collation is searched among detection results of 
the millimeter-wave radar 2. If such previous fusion object is searched, it is considered to have 

15 high probability as being the present fusion object. Accordingly the searched previous fusion 
object is marked. 

[0048] In step S12, calculation weighted on the angle is performed with respect to the 
similarity of each of n i present image objects to the previous fusion objects (n3 pre). It is then 
determined whether there is the image object with high similarity to the present fusion object, 

20 that is, the similarity value equal to or larger than the threshold value. If it is determined there 
is the image object having high similarity, the previous fusion object is marked 

[0049] The process proceeds to step S 1 3 where it is determined whether the 
detennination with respect to all the n3 pre previous fusion objects have been completed. The 
process in step S 12 is repeatedly executed until completion of the determination with respect to 

25 all the previous fusion objects. In the obstruction detection system 1, existence of the previous 
fusion object that has been detected in the previous collation is searched among the detection 
results of the stereo camera 3. If the existence of the previous fusion object is searched, the 
searched previous fusion object is marked as it exhibits high probability of being the present 
fusion object 
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[0050] In step S 14, comparison is made between the millimetric-wave objects collation 
results and the image objects collation results so as to determine the previous fusion objects 
(n3') that have been marked in both the millimetric-wave object and the image object collation 
results as the time-series fusion objects (n3 5 ). Then the infoimation of the time-series fusion 

5 objects (distance, relative speed, angle) is set In the obstruction detection system 1 , results of 
collation between the previous fusion objects and the present millimetric-wave objects, and 
between the previous fusion objects and the present image objects, that is, (millimetric-wave 
objects collation results and image objects collation results) are compared bilaterally. The 
previous fusion object that is marked both in those two collation results is determined as the 

10 object to be detected by the obstruction detection system 1 . 

[0051] The process for executing the instantaneous collation will be described referring to 
the flowchart of Fig. 6. In the case where the time-series fusion object is detected by the time- 
series collation, the time-series fusion objects are excluded from the present millimetric-wave 
objects, that is, n m-n3' millimetric-wave objects are subjected to the instantaneous collation, 

15 and the time-series fusion objects are excluded from the present image objects, that is, n i-n3' 
image objects are subjected to the instantaneous collation. In the case where no time-series 
fusion object is detected, the present millimetric-wave objects and the present image objects are 
subjected to the instantaneous collation. 

[0052] In step S20, each of (n i - n3') image objects is collated with respect to each of the 

20 millimetric-wave objects as the reference respectively in the obstruction detection system 1 . 
Then the image object that is closest to the reference millimetric-wave object is selected If 
each of the difference in the distance and the angle between the reference millimetric-wave 
object and the selected image object is equal to or smaller than each of the respective threshold 
values, the millimetric-wave base pair including the reference millimetric-wave object and the 

25 image object closest thereto is determined. 

[0053] Then in step S21, it is determined whether determination with respect to all the (n 
m-n3') millimetric-wave objects has been completed in the obstruction detection system 1. 
This process in step S20 is repeatedly executed until completion of the aforementioned 
determination. In the obstruction detection system 1, each of detection results of the stereo 
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camera 3 is collated with respect to the millimeter-wave radar 2 so as to search one image 
object that is the highest probability as being closest to the respective millimetric-wave object 
[0054] In step S22, each of (n m - n3') rrallimetric-wave objects is collated with respect 
to each of the image objects as the reference in the obstruction detection system 1 . Then die 
5 . millimetric-wave object that is closest to the reference image object is selected. If each of the 
difference in the distance and the angle between the reference image object and the selected 
millimetric-wave object is equal to or smaller than each of the respective threshold values, the 
image base pair including the reference image object and the millimetric-wave object that is 
closest thereto are determined. 

10 [0055] In step S23, it is determined whether the collation with respect to all the (n i-n3') 

image objects has been completed in the obstruction detection system 1. The process in step 
S22 is repeatedly executed until completion of the collation with respect to all the image 
objects. In the obstruction detection system 1, the detection results of the millimeter-wave 
radar 2 is searched in reference to the detection results of die stereo camera 3 so as to determine 

15 only one millimetric-wave object having highest probability as being closest to the image 
object 

[0056] In step S24, collation is performed between all the millimetric-wave base pairs and 
all the image base pairs that have been determined so as to search the millimetric-wave base 
pair and the image base pair each including the identical millimetric-wave object and the image 
20 object In the obstruction detection system 1 , when the millimetric-wave base pair and the 
image base pair each having the identical millimetric-wave object and the image object are 
searched, the identical millimetric-wave object and the image object are determined as being 
the fusion pair. 

[0057] In step S25, among those fusion pairs, the instantaneous fusion objects (n3") are 
25 determined, and the information of the instantaneous fusion object (distance, relative speed, 
angle) is set In the obstruction detection system 1, comparison is made between the bilateral 
collation results in reference to die millimetric-wave objects and the image objects, that is, 
millimetric-wave base pair and the image base pair. Only when those millimetric-wave base 
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pair and the image base pair have the identical millimetric-wave object and the image object, 
such objects are determined as the fusion objects. 

[0058] In the obstruction detection system 1 , the previous collation result is used for the 
time-series collation. This makes it possible to realize stable collation in comparison with the 
5 . collation between those objects each changes time by time. Even in the case where the object 
to be detected by the obstruction detection system 1 cannot be identified by collating the 
detection results, the above-described process is capable of identifying the object In the case 
where the detection accuracy in the millimeter-wave radar 2 or the stereo camera 3 is 
temporarily deteriorated owing to noise or electric wave hindrance, the collation can be made 

10 with less possibility of foiling to identifying the object to be detected. The similarity is 

determined such that each parameter with higher accuracy of the millimeter-wave radar 2 and 
the stereo camera 3 is weighted. The resultant detection accuracy, thus, is relatively higher. 

[0059] In the instantaneous collation executed in the obstruction detection system 1 , 
millimetric-wave objects and the image objects are bilaterally collated Only in the case of 

15 coincidence in the collation results, it is determined that the fusion object exists. The accuracy 
(fusion accuracy) for determining with respect to coincidence between objects detected by the 
image and detected by the millimetric-wave is substantially high. In the obstruction detection 
system 1, one object either the millimetric-wave object or the image object, which is closest to 
the other object is selected. The collation is performed by searching coincidence among those 

20 pairs of the millimetric-wave base pair and the image pair. The resultant processing load is 
relatively low. In the obstruction detection system 1, the object that has not been detected by 
the previous collation may be identified as being the fusion object in accordance with the 
present detection results. 

[0060] In the obstruction detection system 1 , the detection results of the image of the 

25 vehicle running ahead and the millimetric-wave detection results may be collated so as to 
supply obstruction information having high reliability to various types of operation support 
systems. This makes it possible to support the vehicle operator with appropriate operation. 
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[0061] As has been described with respect to the embodiment of the invention, it is to be 
understood that the invention is not limited to the aforementioned embodiment but may be 
embodied into vanous forms. 

[0062] The embodiment of the invention is applied to the obstruction detection system 
5 equipped for the vehicle. It may be applicable to various types of object detection, for 
example, contact-fiee detection. 

[0063] The embodiment of the invention includes two types of detection units, that is, the 
millimeter-wave radar and the stereo camera However, any other detection unit such as a laser 
radar may be employed. Further three or more detection units may also be employed. 
1 0 [0064] In the embodiment of the invention, the position of each object is identified based 

on the distance, relative speed, and angle. However, other information such as two- 
dimensional coordinate system may be used for identifying the position of the respective 
objects. 

[0065] In the time-series collation according to the embodiment, the object that coincides 
15 in the collations between the previous fusion objects and the present millimetric-wave objects 
and between the previous fusion objects and the present image objects may be set as the time- 
series fusion object The object that is coincided with the previous fusion object in one of those 
collations may be set as the time-series fusion object. 

[0066] While the invention has been described with reference to exemplary embodiments 
20 thereof, it is to be understood that the invention is not limited to the exemplary embodiments or 
constructions. To the contrary, the invention is intended to cover various modifications and 
equivalent arrangements. In addition, while the various elements of the exemplary 
embodiments are shown in various combinations and configurations, which are exemplary, 
other combinations and configurations, including more, less or only a single element, are also 
25 within the spirit and scope of die invention. 



